If two persons wish to have their photographs taken by using a film camera or digital camera while they are standing next to each other, for example, they must use a tripod and a self-timer, or ask a third person such as a stranger to take their photograph.
However, it is a painful job to carry around a tripod. On the other hand, one may feel ill at ease to ask a stranger to take a photograph.
In view of this problem, Japanese Publication for Laid-Open Patent Application, Tokukai 2000-316125 (publication date: Nov. 14, 2000) discloses an image combination device that can extract object regions from a plurality of photographs taken at the same place, and, by combining or not combining an object image with a background, create a background-only image or an image that looks as if an object of another image had been present spontaneously.
Japanese Publication for Laid-Open Patent Application, Tokukai 2001-333327 (publication date: Nov. 30, 2001) discloses a digital camera and an image processing method. With the digital camera and the image processing method, an already photographed image can be used as a reference image, and a specified region (object region) thereof can be displayed on a monitor screen or in a finder as if the specified region were superimposed on an image to be photographed. In addition, it is possible to create image data of a composite image, in which an object in the object region is combined with the image to be photographed.
However, roughly speaking, these prior arts have the following two problems.
The first problem is as follows: if the object region of the reference image is merely carved out and superimposed on another image, (1) an object in the composite image could have a missing part, and/or (2) an unnecessary region could be combined, if the object region is specified imprecisely. In addition, (3) even if the object region is specified precisely, the boundary between combined images could look slightly unnatural.
For example, if (1) the specified object region in the reference image (hereinafter “specified object region”) has a missing part that exists in an actual object region, the part is also missing from the composite image. This makes the composite image look obviously unnatural.
If (2) the specified object region in the reference image is too larger than the actual object region, the specified object region includes a background surrounding the object in the reference image. The “unnecessary region” mentioned above is this background portion. According to the combination method described in Tokukai 2001-333327, an image could be photographed at a place different from the place where the reference image was photographed. Therefore, the background image included in the specified object region (background within the reference image) and the background surrounding the object in the composite image (background within the image to be photographed) could be different. In this case, the composite image looks very unnatural, because the background suddenly changes at the specified object region.
According to the combination method described in Tokukai 2001-333327, even if both images are photographed at the same place with the same background, the specified object region in the reference image could be positioned/combined at an arbitrary position in the image to be photographed. Therefore, the background image included in the specified object region (background within the reference image) and the background, of the image to be photographed, surrounding the position where the two images are combined (background within the image to be photographed) not necessarily backgrounds of the same position. Accordingly, the resultant composite image looks unnatural.
If, as in Tokukai 2001-333327, a user specifies an outline of the specified object region in the reference image by using a tablet or the like, it is unlikely that the specified object region is very imprecise. This is because the outline is traced by the human. However, there is a possibility that errors of one, two, or several pixels might occur. It is not easy to specify the outline by hand with a pixel-size precision.
An example of the case where (3) the boundary between combined images looks slightly unnatural even if the specified object region is precise is a case where, even if the specified object region in (1) and (2) has a pixel-size precision, outline pixels do not match the background of the image to be photographed, as a result of combination of the specified object region.
This is because specifying by pixel is not precise enough to define the outline of the specified object region. To define the outline, a unit finer than a pixel is required. In other words, in an outline pixel, originally (0.X) pixel is an object portion, and (1.0-0.X) pixel is a background portion. The pixel value is a proportional sum (i.e. average value) of the pixel value of the object portion and the pixel value of the background portion.
The proportion between the object portion and the background portion cannot be calculated inversely from the averaged pixel value. Therefore, images must be combined on a pixel-by-pixel basis. As a result, the pixel value of the outline of the composite image includes the value of the background of the reference image, and the outline does not match the surrounding background of the image to be photographed.
The problems (1) to (3) cannot be solved by the combination method disclosed in Tokukai 2000-316125. This publication discloses that positioning is performed before superimposing a plurality of images photographed at the same place or at close-hand positions.
However, if two persons take photographs of each other alternately with the same background, for example, the position of the background appears differently in the photographs taken, due to the difference in camera angles. Besides, the image might be rotated if the camera is inclined; the image might be enlarged or reduced if the distance between the photographer and the object differs; and the image might be distorted if the height of the photographers are different, because the elevation angles of the camera are different in this case.
Therefore, the problems (1) to (3) cannot be solved by merely performing positioning of the images to be superimposed. As a result, the composite image looks unnatural.
The second problem is as follows: if photographs are taken for the purpose of combining an object region of a reference image with an image to be photographed that includes another object, and sufficient attention is not paid to the positions of the objects at the time of shooting, object regions of the respective images might be superimposed on each other in the composite image, or the object of one of the images might extend off the composite image.
In connection with this problem, Tokukai 2000-316125 only describes a combination method using an image that has already been photographed. There is no mention to, for example, a photograph-taking method that can prevent objects from being superimposed on each other or prevent the objects from extending off the composite image.
According to an image-processing method of Tokukai 2001-333327, an object region (the outline of which is specified by the user using a tablet or the like) of a reference image and an image to be photographed can be displayed in a superimposed state. Therefore, in combining the object region of the reference image and the object region of the image to be photographed, it is possible to know, at the time of shooting, whether or not the objects will be superimposed on each other and whether or not an object region will extend off the composite image. If the objects are superimposed or an object extends off, the position of the object in the image to be photographed can be changed by moving the object or the camera. Therefore, it is possible to photograph/record an image in which objects are not superimposed and the objects do not extend off.
However, the foregoing method is inconvenient in that high-level processing such as the processing for recognizing the object regions, the processing for judging whether or not the object regions are superimposed on each other, and the processing for judging whether or not an object region extends off the composite image, must be performed by human. It is also inconvenient that the object region of the reference image must be specified manually.
The present invention was made in view of the foregoing problems. The first objective of the present invention is to provide an image combination device (image combination method) that can combine images without making the composite image look unnatural. The second objective of the present invention is to provide an image combination device (image combination method) that helps photographing so that objects will not be superimposed on each other in a composite image in combining a plurality of objects, which are imaged separately, into a single image as if the objects exist spontaneously.